[UNIX] Is it safe to pipe the output of several parallel processes to one file using >> ?

Posted by conradlee on Stack Overflow See other posts from Stack Overflow or by conradlee
Published on 2010-03-14T21:05:56Z Indexed on 2010/03/14 21:15 UTC
Read the original article Hit count: 225

Filed under:

unix

|

pipe

|

parallel

|

concurrency

I'm scraping data from the web, and I have several processes of my scraper running in parallel.

I want the output of each of these processes to end up in the same file. As long as lines of text remain intact, I the order of the lines does not matter. In UNIX, can I just pipe the output of each process to the same file using the >> operator?

© Stack Overflow or respective owner

Related posts about unix

polkit: disable all users except those in group wheel?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Is it possible to do the following using 1 polkit .pkla file? Disable all users except those in the wheel group from using polkit. The users in the wheel group will need to provide the root password when using polkit. /etc/polkit-1/localauthority/50-local.d/wheel-only.pkla [disable all users… >>> More
How to find other end of unix socket connection?

as seen on Server Fault - Search for 'Server Fault'
I have a process (dbus-daemon) which has many open connection over UNIX sockets. One of these connections is fd #36: =$ ps uw -p 23284 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND depesz 23284 0.0 0.0 24680 1772 ? Ss 15:25 0:00 /bin/dbus-daemon --fork… >>> More
Meaning of directories on Unix and Unix like systems

as seen on Server Fault - Search for 'Server Fault'
I've been using Linux for a couple of years now but I still haven't figured out what the origin or meaning of some the directory names are on Unix and Unix like systems. E.g. what does etc stand for or var? Where does the opt name come from? And while we're on the topic anyway. Can someone give a… >>> More
Easy understanding of UNIX and UNIX shell scripting!!

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, I am beginner to UNIX,UNIX Shell Scripting. Can you please guide me through any excellent sites for unix(which are easy to understand),some study materials(tutorials),video tutorials. Please help!! Thanks! >>> More
Is Mac OS X a licensed Unix or Unix-like clone that conforms to Unix specification?

as seen on Super User - Search for 'Super User'
Is Mac OS X developed on a licensed Unix or is it a Unix-like clone that, unlike Linux, conforms to Unix specification well enough to be registered as a Unix OS. Not until Leopard, Mac OS X did not gain the Unix certification. But in Leopard, Terminal still print: GNU bash, version 3.2.48(1)-release… >>> More

Related posts about pipe

Using SAS Macro to pipe a list of filenames from a Windows directory

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to amend the macro below to accept a macro parameter as the 'location' argument for a dir command. However I cannot get it to resolve correctly due to the nested quotes issue. Using %str(%') does not work, neither do quoting functions for some reason. The macro will work fine when the… >>> More
How to rate-limit a pipe under linux ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there a filter which I could use to rate-limit a pipe on linux? If this exists, let call it rate-limit, I want to be able to type in a terminal something like cat /dev/urandom | rate-limit 3 -k | foo in order to send a a stream of random bytes to foo's standard input at a rate (lower than) 3… >>> More
Behavior of a pipe after a fork()

as seen on Stack Overflow - Search for 'Stack Overflow'
When reading about pipes in Advanced Programming in the UNIX Environment, I noticed that after a fork that the parent can close() the read end of a pipe and it doesn't close the read end for the child. When a process forks, does its file descriptors get retained? What I mean by this is that before… >>> More
How do I pipe terminal standard output (stdout) to the clipboard?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
For example, Say I want to list the contents of a folder and directly paste them into a chat window for a friend to see. I realize I could do ls > filename.txt to create a file (filename.txt) with those contents; I'd then have to open or print the file and manually select and copy the text… >>> More
I can't upgrade 13.10 because of broken pipe

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I try upgrading and this is what I get: christopher@chris-computer:~$ sudo apt-get upgrade [sudo] password for christopher: Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: librhythmbox-core7 1 upgraded, 0… >>> More